Development and evaluation of trigger tools to identify pediatric blood management errors.

BACKGROUND: Pediatric patient blood management (PBM) programs require continuous surveillance of errors and near misses. However, most PBM programs rely on passive surveillance methods. Our objective was to develop and evaluate a set of automated trigger tools for active surveillance of pediatric PBM errors. MATERIALS AND METHODS: We used the Rand-UCLA method with an expert panel of pediatric transfusion medicine specialists to identify and prioritize candidate trigger tools for all transfused blood products. We then iteratively developed automated queries of electronic health record (EHR) data for the highest priority triggers. Two physicians manually reviewed a subset of cases meeting trigger tool criteria and estimated each trigger tool's positive predictive value (PPV). We then estimated the rate of PBM errors, whether they reached the patient, and adverse events for each trigger tool across four years in a single pediatric health system. RESULTS: We identified 28 potential triggers for pediatric PBM errors and developed 5 automated trigger tools (positive patient identification, missing irradiation, unwashed products despite prior anaphylaxis, transfusion lasting >4 hours, over-transfusion by volume). The PPV for ordering errors ranged from 38-100%. The most frequently detected near miss event reaching patients was first transfusions without positive patient identification (estimate 303, 95% CI: 288-318 per year). The only adverse events detected were from over-transfusions by volume, including 4 adverse events detected on manual review that had not been reported in passive surveillance systems. DISCUSSION: It is feasible to automatically detect pediatric PBM errors using existing data captured in the EHR that enable active surveillance systems. Over-transfusions may be one of the most frequent causes of harm in the pediatric environment.

Clinical Decision Support Principles for Quality Improvement and Research.

Pediatric hospitalists frequently interact with clinical decision support (CDS) tools in patient care and use these tools for quality improvement or research. In this method/ology paper, we provide an introduction and practical approach to developing and evaluating CDS tools within the electronic health record. First, we define CDS and describe the types of CDS interventions that exist. We then outline a stepwise approach to CDS development, which begins with defining the problem and understanding the system. We present a framework for metric development and then describe tools that can be used for CDS design (eg, 5 Rights of CDS, "10 commandments," usability heuristics, human-centered design) and testing (eg, validation, simulation, usability testing). We review approaches to evaluating CDS tools, which range from randomized studies to traditional quality improvement methods. Lastly, we discuss practical considerations for implementing CDS, including the assessment of a project team's skills and an organization's information technology resources.

Influence of user-centered clinical decision support on pediatric blood product ordering errors.

BACKGROUND: Children are at increased risk from transfusion-related medical errors. Clinical decision support (CDS) can enhance pediatric providers' decision-making regarding transfusion practices including indications, volume, rate, and special processing instructions. Our objective was to use CDS in a pediatric health system to reduce:blood product-related safety events from ordering errors;special processing ordering errors for patients with T-cell dysfunction, sickle cell disease (SCD), or thalassemia;transfusions administered faster than 5 mL/kg/h. MATERIALS AND METHODS: In this single-center before and after quality improvement study, we evaluated how user-centered design of pediatric blood product orders influenced pediatric transfusion practices and outcomes. Safety events were identified through active and passive surveillance. Other clinically relevant outcomes were identified through electronic health record queries. RESULTS: Blood product-related safety events from ordering errors did not change significantly from the baseline period (6 events, 0.4 per month, from 1/1/2018-3/27/2019) to the intervention period (1 event, 0.1 per month, from 3/28/2019-12/31/2019; rate ratio: 0.27 [0.01-2.25]). Packed red blood cell (PRBC) and platelet orders for patients with T-cell dysfunction that did not specify irradiation decreased significantly from 488/12,359 (3.9%) to 204/6,711 (3.0%, risk ratio: 0.77 [0.66-0.90]). PRBC orders for patients with SCD or thalassemia that did not specify phenotypically similar units fell from 386/2,876 (13.4%) to 57/1,755 (3.2%, risk ratio: 0.24 [0.18-0.32]). Transfusions administered faster than 5 mL/kg/h decreased from 4,112/14,641 (28.1%) to 2,125/9,263 (22.9%, risk ratio: 0.82 [0.78-0.85]). DISCUSSION: User-centered design of CDS for pediatric blood product orders significantly reduced special processing ordering errors and inappropriate transfusion rates. Larger studies are needed to evaluate the impact on safety events.

Integrating structured and unstructured data for timely prediction of bloodstream infection among children.

BACKGROUND: Hospitalized children with central venous lines (CVLs) are at higher risk of hospital-acquired infections. Information in electronic health records (EHRs) can be employed in training deep learning models to predict the onset of these infections. We incorporated clinical notes in addition to structured EHR data to predict serious bloodstream infections, defined as positive blood culture followed by at least 4 days of new antimicrobial agent administration, among hospitalized children with CVLs. METHODS: Structured EHR information and clinical notes were extracted for a retrospective cohort including all hospitalized patients with CVLs at a single tertiary care pediatric health system from 2013 to 2018. Deep learning models were trained to determine the added benefit of incorporating the information embedded in clinical notes in predicting serious bloodstream infection. RESULTS: A total of 24,351 patient encounters met inclusion criteria. The best-performing model restricted to structured EHR data had a specificity of 0.951 and positive predictive value (PPV) of 0.056 when the sensitivity was set to 0.85. The addition of contextualized word embeddings improved the specificity to 0.981 and PPV to 0.113. CONCLUSIONS: Integrating clinical notes with structured EHR data improved the prediction of serious bloodstream infections among pediatric patients with CVLs. IMPACT: Developed an advanced infection prediction model in pediatrics that integrates the structured and unstructured EHRs. Extracted information from clinical notes to do timely prediction in a clinical setting. Developed a deep learning model framework that can be employed in predicting rare events in a complex and dynamic environment.

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts.

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Automated Identification of Immunocompromised Status in Critically Ill Children.

BACKGROUND: Easy identification of immunocompromised hosts (ICHs) would allow for stratification of culture results based on host type. METHODS: We utilized antimicrobial stewardship program (ASP) team notes written during handshake stewardship rounds in the pediatric intensive care unit (PICU) as the gold standard for host status; clinical notes from the primary team, medication orders during the encounter, problem list, and billing diagnoses documented prior to the ASP documentation were extracted to develop models that predict host status. We calculated performance for three models based on diagnoses/medications, with and without natural language processing from clinical notes. The susceptibility of pathogens causing bacteremia to commonly used empiric antibiotic regimens was then stratified by host status. RESULTS: We identified 844 antimicrobial episodes from 666 unique patients; 160 (18.9%) were identified as ICHs. We randomly selected 675 initiations (80%) for model training and 169 initiations (20%) for testing. A rule-based model using diagnoses and medications alone yielded a sensitivity of 0.87 (08.6-0.88), specificity of 0.93 (0.92-0.93), and positive predictive value (PPV) of 0.74 (0.73-0.75). Adding clinical notes into XGBoost model led to improved specificity of 0.98 (0.98-0.98) and PPV of 0.9 (0.88-0.91), but with decreased sensitivity 0.77 (0.76-0.79). There were 77 bacteremia episodes during the study period identified and a host-specific visualization was created. CONCLUSIONS: An electronic health record-based phenotype based on notes, diagnoses, and medications identifies ICH in the PICU with high specificity.

Lessons Learned from OpenNotes Learning Mode and Subsequent Implementation across a Pediatric Health System.

BACKGROUND: The 21st Century Cures Act has accelerated adoption of OpenNotes, providing new opportunities for patient and family engagement in their care. However, these regulations present new challenges, particularly for pediatric health systems aiming to improve information sharing while minimizing risks associated with adolescent confidentiality and safety. OBJECTIVE: Describe lessons learned preparing for OpenNotes across a pediatric health system during a 4-month trial period (referred to as "Learning Mode") in which clinical notes were not shared by default but decision support was present describing the upcoming change and physicians could request feedback on complex cases from a multidisciplinary team. METHODS: During Learning Mode (December 3, 2020-March 9, 2021), implementation included (1) educational text at the top of commonly used note types indicating that notes would soon be shared and providing guidance, (2) a new confidential note type, and (3) a mechanism for physicians to elicit feedback from a multidisciplinary OpenNotes working group for complex cases with questions related to OpenNotes. The working group reviewed lessons learned from this period, as well as implementation of OpenNotes from March 10, 2021 to June 30, 2021. RESULTS: During Learning Mode, 779 confidential notes were written across the system. The working group provided feedback on 14 complex cases and also reviewed 7 randomly selected confidential notes. The proportion of physician notes shared with patients increased from 1.3% to 88.4% after default sharing of notes to the patient portal. Key lessons learned included (1) sensitive information was often present in autopopulated elements, differential diagnoses, and supervising physician note attestations; and (2) incorrect reasons were often selected by clinicians for withholding notes but this accuracy improved with new designs. CONCLUSION: While OpenNotes provides an unprecedented opportunity to engage pediatric patients and their families, targeted education and electronic health record designs are needed to mitigate potential harms of inappropriate disclosures.

Deep Learning Model to Predict Serious Infection Among Children With Central Venous Lines.

Objective: Predict the onset of presumed serious infection, defined as a positive blood culture drawn and new antibiotic course of at least 4 days (PSI*), among pediatric patients with Central Venous Lines (CVLs). Design: Retrospective cohort study. Setting: Single academic children's hospital. Patients: All hospital encounters from January 2013 to December 2018, excluding the ones without a CVL or with a length-of-stay shorter than 24 h. Measurements and Main Results: Clinical features including demographics, laboratory results, vital signs, characteristics of the CVLs and medications used were extracted retrospectively from electronic medical records. Data were aggregated across all hospitals within a single pediatric health system and used to train a deep learning model to predict the occurrence of PSI* during the next 48 h of hospitalization. The proposed model prediction was compared to prediction of PSI* by a marker of illness severity (PELOD-2). The baseline prevalence of line infections was 0.34% over all segmented 48-h time windows. Events were identified among cases using onset time. All data from admission till the onset was used for cases and among controls we used all data from admission till discharge. The benchmarks were aggregated over all 48 h time windows [N=748,380 associated with 27,137 patient encounters]. The model achieved an area under the receiver operating characteristic curve of 0.993 (95% CI = [0.990, 0.996]), the enriched positive predictive value (PPV) was 23 times greater than the base prevalence. Conversely, prediction by PELOD-2 achieved a lower PPV of 1.5% [0.9%, 2.1%] which was 5 times the baseline prevalence. Conclusion: A deep learning model that employs common clinical features in the electronic health record can help predict the onset of CLABSI in hospitalized children with central venous line 48 hours prior to the time of specimen collection.

Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics.

BACKGROUND: Excessive electronic health record (EHR) alerts reduce the salience of actionable alerts. Little is known about the frequency of interruptive alerts across health systems and how the choice of metric affects which users appear to have the highest alert burden. OBJECTIVE: (1) Analyze alert burden by alert type, care setting, provider type, and individual provider across 6 pediatric health systems. (2) Compare alert burden using different metrics. MATERIALS AND METHODS: We analyzed interruptive alert firings logged in EHR databases at 6 pediatric health systems from 2016-2019 using 4 metrics: (1) alerts per patient encounter, (2) alerts per inpatient-day, (3) alerts per 100 orders, and (4) alerts per unique clinician days (calendar days with at least 1 EHR log in the system). We assessed intra- and interinstitutional variation and how alert burden rankings differed based on the chosen metric. RESULTS: Alert burden varied widely across institutions, ranging from 0.06 to 0.76 firings per encounter, 0.22 to 1.06 firings per inpatient-day, 0.98 to 17.42 per 100 orders, and 0.08 to 3.34 firings per clinician day logged in the EHR. Custom alerts accounted for the greatest burden at all 6 sites. The rank order of institutions by alert burden was similar regardless of which alert burden metric was chosen. Within institutions, the alert burden metric choice substantially affected which provider types and care settings appeared to experience the highest alert burden. CONCLUSION: Estimates of the clinical areas with highest alert burden varied substantially by institution and based on the metric used.

Evaluation of a Clinical Decision Support Strategy to Increase Seasonal Influenza Vaccination Among Hospitalized Children Before Inpatient Discharge.

Importance: Hospitalized children are at increased risk of influenza-related complications, yet influenza vaccine coverage remains low among this group. Evidence-based strategies about vaccination of vulnerable children during all health care visits are especially important during the COVID-19 pandemic. Objective: To design and evaluate a clinical decision support (CDS) strategy to increase the proportion of eligible hospitalized children who receive a seasonal influenza vaccine prior to inpatient discharge. Design, Setting, and Participants: This quality improvement study was conducted among children eligible for the seasonal influenza vaccine who were hospitalized in a tertiary pediatric health system providing care to more than half a million patients annually in 3 hospitals. The study used a sequential crossover design from control to intervention and compared hospitalizations in the intervention group (2019-2020 season with the use of an intervention order set) with concurrent controls (2019-2020 season without use of an intervention order set) and historical controls (2018-2019 season with use of an order set that underwent intervention during the 2019-2020 season). Interventions: A CDS intervention was developed through a user-centered design process, including (1) placing a default influenza vaccine order into admission order sets for eligible patients, (2) a script to offer the vaccine using a presumptive strategy, and (3) just-in-time education for clinicians addressing vaccine eligibility in the influenza order group with links to further reference material. The intervention was rolled out in a stepwise fashion during the 2019-2020 influenza season. Main Outcomes and Measures: Proportion of eligible hospitalizations in which 1 or more influenza vaccines were administered prior to discharge. Results: Among 17 740 hospitalizations (9295 boys [52%]), the mean (SD) age was 8.0 (6.0) years, and the patients were predominantly Black (n = 8943 [50%]) or White (n = 7559 [43%]) and mostly had public insurance (n = 11 274 [64%]). There were 10 997 hospitalizations eligible for the influenza vaccine in the 2019-2020 season. Of these, 5449 (50%) were in the intervention group, and 5548 (50%) were concurrent controls. There were 6743 eligible hospitalizations in 2018-2019 that served as historical controls. Vaccine administration rates were 31% (n = 1676) in the intervention group, 19% (n = 1051) in concurrent controls, and 14% (n = 912) in historical controls (P < .001). In adjusted analyses, the odds of receiving the influenza vaccine were 3.25 (95% CI, 2.94-3.59) times higher in the intervention group and 1.28 (95% CI, 1.15-1.42) times higher in concurrent controls than in historical controls. Conclusions and Relevance: This quality improvement study suggests that user-centered CDS may be associated with significantly improved influenza vaccination rates among hospitalized children. Stepwise implementation of CDS interventions was a practical method that was used to increase quality improvement rigor through comparison with historical and concurrent controls.

Cost-effectiveness of infant respiratory syncytial virus preventive interventions in Mali: A modeling study to inform policy and investment decisions.

IMPORTANCE: Low- and middle-income countries have a high burden of respiratory syncytial virus lower respiratory tract infections. A monoclonal antibody administered monthly is licensed to prevent these infections, but it is cost-prohibitive for most low- and middle-income countries. Long-acting monoclonal antibodies and maternal vaccines against respiratory syncytial virus are under development. OBJECTIVE: We estimated the likelihood of respiratory syncytial virus preventive interventions (current monoclonal antibody, long-acting monoclonal antibody, and maternal vaccine) being cost-effective in Mali. DESIGN: We modeled age-specific and season-specific risks of respiratory syncytial virus lower respiratory tract infections within monthly cohorts of infants from birth to six months. We parameterized with respiratory syncytial virus data from Malian cohort studies, as well as product efficacy from clinical trials. Integrating parameter uncertainty, we simulated health and economic outcomes for status quo without prevention, intra-seasonal monthly administration of licensed monoclonal antibody, pre-seasonal birth dose administration of a long-acting monoclonal antibody, and maternal vaccination. We then calculated the incremental cost-effectiveness ratio of each intervention compared to status quo from the perspectives of the government, donor, and society. RESULTS: At a price of $3 per dose and from the societal perspective, current monoclonal antibody, long-acting monoclonal antibody, and maternal vaccine would have incremental cost-effectiveness ratios of $4280 (95% CI $1892 to $122,434), $1656 (95% CI $734 to $9091), and $8020 (95% CI $3501 to $47,047) per disability-adjusted life-year averted, respectively. CONCLUSIONS AND RELEVANCE: In Mali, long-acting monoclonal antibody is likely to be cost-effective from both the government and donor perspectives at $3 per dose. Maternal vaccine would need higher efficacy over that measured by a recent trial in order to be considered cost-effective.

Predicting presumed serious infection among hospitalized children on central venous lines with machine learning.

BACKGROUND: Presumed serious infection (PSI) is defined as a blood culture drawn and new antibiotic course of at least 4 days among pediatric patients with Central Venous Lines (CVLs). Early PSI prediction and use of medical interventions can prevent adverse outcomes and improve the quality of care. METHODS: Clinical features including demographics, laboratory results, vital signs, characteristics of the CVLs and medications used were extracted retrospectively from electronic medical records. Data were aggregated across all hospitals within a single pediatric health system and used to train machine learning models (XGBoost and ElasticNet) to predict the occurrence of PSI 8 h prior to clinical suspicion. Prediction for PSI was benchmarked against PRISM-III. RESULTS: Our model achieved an area under the receiver operating characteristic curve of 0.84 (95% CI = [0.82, 0.85]), sensitivity of 0.73 [0.69, 0.74], and positive predictive value (PPV) of 0.36 [0.34, 0.36]. The PRISM-III conversely achieved a lower sensitivity of 0.19 [0.16, 0.22] and PPV of 0.30 [0.26, 0.34] at a cut-off of ≥ 10. The features with the most impact on the PSI prediction were maximum diastolic blood pressure prior to PSI prediction (mean SHAP = 3.4), height (mean SHAP = 3.2), and maximum temperature prior to PSI prediction (mean SHAP = 2.6). CONCLUSION: A machine learning model using common features in the electronic medical records can predict the onset of serious infections in children with central venous lines at least 8 h prior to when a clinical team drew a blood culture.

Quality Initiative to Reduce High-Flow Nasal Cannula Duration and Length of Stay in Bronchiolitis.

OBJECTIVES: High-flow nasal cannula (HFNC) use in bronchiolitis may prolong length of stay (LOS) if weaned more slowly than medically indicated. We aimed to reduce HFNC length of treatment (LOT) and inpatient LOS by 12 hours in 0- to 18-month-old patients with bronchiolitis on the pediatric hospital medicine service. METHODS: After identifying key drivers of slow weaning, we recruited a multidisciplinary "Wean Team" to provide education and influence provider weaning practices. We then implemented a respiratory therapist-driven weaning protocol with supportive sociotechnical interventions (huddles, standardized orders, simplification of protocol) to reduce LOT and LOS and promote sustainability. RESULTS: In total, 283 patients were included: 105 during the baseline period and 178 during the intervention period. LOT and LOS control charts revealed special cause variation at the start of the intervention period; mean LOT decreased from 48.2 to 31.2 hours and mean LOS decreased from 84.3 to 60.9 hours. LOT and LOS were less variable in the intervention period compared with the baseline period. There was no increase in PICU transfers or 72-hour return or readmission rates. CONCLUSIONS: We reduced HFNC LOT by 17 hours and LOS by 23 hours for patients with bronchiolitis via multidisciplinary collaboration, education, and a respiratory therapist-driven weaning protocol with supportive interventions. Future steps will focus on more judicious application of HFNC in bronchiolitis.

Reducing Prescribing Errors in Hospitalized Children on the Ketogenic Diet.

BACKGROUND: Children on the ketogenic diet must limit carbohydrate intake to maintain ketosis and reduce seizure burden. Patients on ketogenic diet are vulnerable to harm in the hospital setting where carbohydrate-containing medications are commonly prescribed. We developed clinical decision support to reduce inappropriate prescription of carbohydrate-containing medications in hospitalized children on ketogenic diet. METHODS: A clinical decision support alert was developed through formative and summative usability testing. The alert warned prescribers when they entered an order for a carbohydrate-containing medication in patients on ketogenic diet. The alert was implemented using a quasi-experimental design with sequential crossover from control to intervention at two tertiary care pediatric hospitals within a single health system. The primary outcome was carbohydrate-containing medication orders per patient-day. RESULTS: During the study period, there were 280 ketogenic diet patient admissions totaling 1219 patient-days. The carbohydrate-containing medication order rate declined from 0.69 to 0.35 orders per patient-day (absolute rate reduction 0.34, 95% confidence interval 0.25-0.43), corresponding to 256 inappropriate orders prevented. The alert fired 398 times and was accepted (i.e., the order was removed) 227 times for an overall acceptance rate of 57%. CONCLUSIONS: Implementation of a clinical decision support alert at order-entry resulted in a sustained reduction in carbohydrate-containing medication orders for hospitalized patients on ketogenic diet without an increase in alert burden. Clinical decision support developed with user-centered design principles can improve patient safety for children on ketogenic diet by influencing prescriber behavior.

Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs.

OBJECTIVE: Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider-patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs). METHODS: We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow. RESULTS: We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19-278.56; ED: OR = 19.06, 95% CI: 9.53-41.65' inpatient: OR = 2.95, 95% CI: 2.23-3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%). CONCLUSION: Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.

Pediatric trainees systematically under-report duty hour violations compared to electronic health record defined shifts.

Duty hour monitoring is required in accredited training programs, however trainee self-reporting is onerous and vulnerable to bias. The objectives of this study were to use an automated, validated algorithm to measure duty hour violations of pediatric trainees over a full academic year and compare to self-reported violations. Duty hour violations calculated from electronic health record (EHR) logs varied significantly by trainee role and rotation. Block-by-block differences show 36.8% (222/603) of resident-blocks with more EHR-defined violations (EDV) compared to self-reported violations (SRV), demonstrating systematic under-reporting of duty hour violations. Automated duty hour tracking could provide real-time, objective assessment of the trainee work environment, allowing program directors and accrediting organizations to design and test interventions focused on improving educational quality.

Formative Usability Testing Reduces Severe Blood Product Ordering Errors.

BACKGROUND: Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process. OBJECTIVES: There are two objectives of this study as follows:(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD. METHODS: A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample t-tests. RESULTS: Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9-57%; p = 0.03). Time on task did not vary significantly. CONCLUSION: UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design.

Towards a Maturity Model for Clinical Decision Support Operations.

Clinical decision support (CDS) systems delivered through the electronic health record are an important element of quality and safety initiatives within a health care system. However, managing a large CDS knowledge base can be an overwhelming task for informatics teams. Additionally, it can be difficult for these informatics teams to communicate their goals with external operational stakeholders and define concrete steps for improvement. We aimed to develop a maturity model that describes a roadmap toward organizational functions and processes that help health care systems use CDS more effectively to drive better outcomes. We developed a maturity model for CDS operations through discussions with health care leaders at 80 organizations, iterative model development by four clinical informaticists, and subsequent review with 19 health care organizations. We ceased iterations when feedback from three organizations did not result in any changes to the model. The proposed CDS maturity model includes three main "pillars": "Content Creation," "Analytics and Reporting," and "Governance and Management." Each pillar contains five levels-advancing along each pillar provides CDS teams a deeper understanding of the processes CDS systems are intended to improve. A "roof" represents the CDS functions that become attainable after advancing along each of the pillars. Organizations are not required to advance in order and can develop in one pillar separately from another. However, we hypothesize that optimal deployment of preceding levels and advancing in tandem along the pillars increase the value of organizational investment in higher levels of CDS maturity. In addition to describing the maturity model and its development, we also provide three case studies of health care organizations using the model for self-assessment and determine next steps in CDS development.

Development and dissemination of clinical decision support across institutions: standardization and sharing of refugee health screening modules.

OBJECTIVES: We developed and piloted a process for sharing guideline-based clinical decision support (CDS) across institutions, using health screening of newly arrived refugees as a case example. MATERIALS AND METHODS: We developed CDS to support care of newly arrived refugees through a systematic process including a needs assessment, a 2-phase cognitive task analysis, structured preimplementation testing, local implementation, and staged dissemination. We sought consensus from prospective users on CDS scope, applicable content, basic supported workflows, and final structure. We documented processes and developed sharable artifacts from each phase of development. We publically shared CDS artifacts through online dissemination platforms. We collected feedback and implementation data from implementation sites. RESULTS: Responses from 19 organizations demonstrated a need for improved CDS for newly arrived refugee patients. A guided multicenter workflow analysis identified 2 main workflows used by organizations that would need to be supported by shared CDS. We developed CDS through an iterative design process, which was successfully disseminated to other sites using online dissemination repositories. Implementation sites had a small-to-modest analyst time commitment but reported a good match between CDS and workflow. CONCLUSION: Sharing of CDS requires overcoming technical and workflow barriers. We used a guided multicenter workflow analysis and online dissemination repositories to create flexible CDS that has been adapted at 3 sites. Organizations looking to develop sharable CDS should consider evaluating the workflows of multiple institutions and collecting feedback on scope, design, and content in order to make a more generalizable product.

Automatic Detection of Front-Line Clinician Hospital Shifts: A Novel Use of Electronic Health Record Timestamp Data.

OBJECTIVE: Excess physician work hours contribute to burnout and medical errors. Self-report of work hours is burdensome and often inaccurate. We aimed to validate a method that automatically determines provider shift duration based on electronic health record (EHR) timestamps across multiple inpatient settings within a single institution. METHODS: We developed an algorithm to calculate shift start and end times for inpatient providers based on EHR timestamps. We validated the algorithm based on overlap between calculated shifts and scheduled shifts. We then demonstrated a use case by calculating shifts for pediatric residents on inpatient rotations from July 1, 2015 through June 30, 2016, comparing hours worked and number of shifts by rotation and role. RESULTS: We collected 6.3 × 107 EHR timestamps for 144 residents on 771 inpatient rotations, yielding 14,678 EHR-calculated shifts. Validation on a subset of shifts demonstrated 100% shift match and 87.9 ± 0.3% overlap (mean ± standard error [SE]) with scheduled shifts. Senior residents functioning as front-line clinicians worked more hours per 4-week block (mean ± SE: 273.5 ± 1.7) than senior residents in supervisory roles (253 ± 2.3) and junior residents (241 ± 2.5). Junior residents worked more shifts per block (21 ± 0.1) than senior residents (18 ± 0.1). CONCLUSION: Automatic calculation of inpatient provider work hours is feasible using EHR timestamps. An algorithm to assess provider work hours demonstrated criterion validity via comparison with scheduled shifts. Differences between junior and senior residents in calculated mean hours worked and number of shifts per 4-week block were also consistent with differences in scheduled shifts and duty-hour restrictions.